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Description 

The present invention relates to video image signal processing tor storage or transmissibn systems in which motion 
vectors are calculated to allow subsequent interpolation either of additional lines in each field or of additional fields. 

5 Many algorithms for motion estimation have been developed. One of these is disclosed in an article "New algorithm 

for motion estimation" by Gerard de Haan and Henk Huijgen, Proceedings of the Third International Workshop on HDTV, 
Torino, 1 989. US-A-4,81 6,91 4 discloses a method and apparatus for efficiently encoding and decoding image sequences, 
in which motion vectors are determined for each of a plurality of image blocks together constituting a television image 
with first means for computing motion vectors for image blocks, and second means for interpolating motion vectors for 

10 individual pixels based on computed motion vectors of the blocks. 

It is inter alia an object of the present invention to simplify the motion estimator hardware and/or decrease the data 
transmission/storage capacity requirement, e.g. in a system in which a television field has been divided into a plurality 
of blocks and it is required to calculate motion vectors to each of these blocks. 

One aspect of the present invention provides a method or system in which motion vectors are computed for selected 

is ones of the blocks, the vectors for the remaining blocks being derived by interpolation from the computed motion vectors. 
The invention is set out more precisely in the annexed claims 1,6, and 10. Advantageous embodiments are defined in 
the subclaims. 

Additional aspects, advantages and details of the manner of operation of the present invention will become clear 
with reference to the following description taken in conjunction with the drawings. 
20 in the drawings: 



Fig. 1 illustrates a first pattern of interleaving blocks for which a motion vector is calculated with those for which a 
motion vector will be interpolated at the display end; 
Fig. 2 illustrates an alternate pattern for that of Fig. 1 ; 
25 Fig. 3 is an expanded view of one section of Fig. 1 ; 

Fig. 4 is a block diagram of a delay structure for component interpolation in the horizontal direction; 
Fig. 5 is a block diagram of a median motion vector component calculating circuit; 
Fig. 6 is a block diagram of a delay structure for component interpolation in the vertical direction; and 
Fig. 7 is a block diagram of the delay structure for cascaded horizontal-vertical component interpolation. 

30 

The terms "television" and "video" are used interchangeably throughout this application. It will be appreciated that 
the invention is applicable in other than strictly defined "television" fields (e.g. conferencing systems). Similarly, although 
"odd" and "even fields" are mentioned, the invention is also applicable to sequential scan systems in which the sequential 
"fields" are, technically, frames. 

35 As mentioned above, the present invention concerns itself with video systems in which the image is subdivided into 

a number of blocks. In Fig. 1 , in known systems, this subdivision would include all the blocks, both those marked "X" 
and those marked "O". For each of these blocks a motion vector would be calculated. The calculation as such is not 
part of the present invention. The only assumptions made are that a motion vector is assigned to each block, that the 
calculation of such motion vectors is relatively complicated and that storage and/or transmission of such vectors requires 

40 corresponding storage anoVor transmission capacities. 

The present invention causes a reduction in hardware, computation time, and storage/transmission capacity by 
requiring that only half of the number of vectors is calculated at the transmitting or recording end and then stored and/or 
transmitted, or, alternatively, calculated at the receiving or display end. The remaining vectors, according to the present 
invention, are calculated with a simple algorithm from the vectors associated with the other blocks. 

45 The present invention will be discussed in detail with regard to the interleaving scheme or pattern of Fig. 1, where 

"X" indicates the blocks for which the vector is calculated in odd fields, while "O" indicates the blocks for which the vector 
is calculated in even fields. Obviously, these could be reversed or a completely different scheme, for example that shown 
in Fig. 2, could be used. A reason for discussion of the scheme in Fig. 1 at this time is that simulation has shown that 
the overall image display quality of the system is virtually unchanged for this particular pattern. Further, the problem of 

50 interpolation of the missing motion vectors for the scheme of Fig. 2 is very similar to the problem of progressive scan 
conversion of an interlaced television signal. The many known techniques for progressive scan conversion can thus 
readily be applied here as well, so that no further discussion of the Fig. 2 system is required. 

Returning now to Fig. 1 , in a preferred embodiment vectors are assigned to blocks for which no vector was calculated 
on the following basis. The x and y components are determined separately by selecting the median value of the corre- 

55 sponding component in at least two adjacent blocks and the corresponding component in the same block in the previous 
field. For the following discussion, reference should also be made to fig. 3, in which letters are assigned to each block 
in the vicinity of a block H for which the x and y components of a motion vector are to be found. The delays in figs. 4, 6 
and 7 are marked t for a block delay, T for a one block field delay, and T L for a one block line delay. For deriving the 
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missing vectors by median interpolation utilizing the two blocks, E and K, directly adjacent in the horizontal direction, 
the delay hardware illustrated in fig. 4 may be used. If the interpolation is for the x-components of the motion vectors, 
the signals supplied to terminal 10 are the computed E and K x-components. 

If it is assumed that the incoming component is the x-component of the K motion vector, then K is available at output 

s terminal 1 1 . Input terminal 1 0 is also connected to a one block delay 1 2 which, in turn, is connected to a one block delay 
14 and a one block field delay 16. At a second output terminal 13, connected to the output of delay stage 14, the 
x-component of the motion vector associated with block E is present, while at a terminal 15 at the output of block field 
delay 16 the x-component of the H-vector of the previous field will be available. The H-vector of the previous field is 
indicated by H(-1 ). These values may be applied to comparators 20, 22 and 24 as indicated in fig. 5. The sign output of 

10 each comparator is applied to a look-up table (LUT) 26 whose output controls a multiplexer (MUX) 28. In accordance 
with the control signals furnished at the output of look-up table 26, multiplexer 28 will select the median value among 
E, K, and H(-1 ). This is indicated as R at the output of multiplexer 28. This value alone could be used as the x-component 
for the motion vector associated with block H in fig. 3. The same circuit as that shown in fig. 4 would be utilized to derive 
the y-component. 

15 Alternatively, the x- and y-components of the motion vector to be assigned to block H could be derived from the 

corresponding components in vertically adjacent blocks I and G, as well as the value of the vector component assigned 
to block H in the previous field. The relevant delay structures are illustrated in fig. 6. The x- (or y-) components of the 
motion vectors are received at a terminal 30. If it is assumed that at the particular time instant the motion vector com- 
ponent corresponding to block I is received, then one block line delay 32 plus one block field delay 34 later, the vector 

20 value assigned to block H in the preceding field will be available. Finally, the value of the motion vector component 
associated with block G will be available at the output of one block line delay 36. If the values of the motion vector 
component associated with blocks I and G were substituted for those indicated by E and K in fig. 5, the output R of 
multiplexer (MUX) 28 would be another estimate for the x-component (or the y-component) of the motion vector asso- 
ciated with block H. Either the interpolation scheme illustrated in fig. 4 or that is illustrated in fig. 6 could be used. 

2S Further image improvement would result if, instead of arbitrarily selecting either the horizontal or the vertical inter- £"* 

polation scheme as illustrated in figs. 4 and 6, respectively, a decision between vertical or horizontal interpolation were "Zk 
made on the basts of changes in the vector field. For example, the median could be taken in the direction with a smallest ' 
difference between the vectors calculated for the two adjacent blocks. If the difference is larger in the vertical direction, . - .» 1st 

a horizontal interpolation would be carried out and vice versa. iv-r*-* 

30 However, best results have been obtained by use of the complete circuit in fig. 5. For this purpose, the delays 

illustrated in figs. 4 and 6 would be combined as shown in fig. 7. Here, the motion vector component, either the x-com- 
ponent or the y-component, is received at a terminal 40. The component undergoes a delay of one block line minus one v 
block in a delay stage 42. The output of stage 42 is the vector associated with block K. After a further block delay in a *4S 
stage 44, the motion vector component signals are applied to a block line delay 46 which therefore yields the motion — 

35 vector associated with block G at its output. The output of delay 44 is further applied to an additional block delay 48 

whose output therefore yields the motion vector associated with block E. Finally, the output of delay block 44 is applied- *ave*e 
to a block field delay 50 whose output is the value of the motion vector associated with the H block of the previous field. 

All of the above mentioned signals, i.e. motion vector components, are applied to correspondingly designated ter- 
minals in fig. 5. The motion vectors associated with blocks E, K and H(-1 ) are applied to comparators 20, 22 and 24 as 

40 described previously. The motion vectors associated with blocks G and I are applied, again as indicated in fig. 5, to 
comparators 30, 32 and 34. The second input of comparators 32 and 34 is the signal R at the output of multiplexer 28, 
i.e. the median value found by interpolation in the x-direction. These sign signals at the outputs of comparators 30, 32 
and 34 are applied to a second look-up table 36 which controls a multiplexer 38 to output the median value of G, I and 
R. In this last embodiment, the vector value for block H in the present field was thus computed from the vector values 

45 in the 4 neighboring blocks E, K, G and I and the vector in the former field at the position of block H. The circuits of fig. 
7 and 5 for this last embodiment of the present invention will be required twice, once for the x-component of the vectors, 
and once for the y-component of the vectors. In addition, the computed values of the motion vectors and the interpolated 
values have to be interleaved either directly by multiplexers or upon readout from memory. 

From the above discussion it will be clear that above aspects of the present invention, are entirely independent of 

50 the particular algorithms by which motion vectors may be derived and that the calculation of motion vectors for selected 
blocks only and the interpolation of motion vectors of the remaining blocks from values derived from blocks having 
calculated motion vectors may be implemented in many different ways which will readily be apparent to one skilled in 
the art. A preferred motion estimation method is discussed below. 

The processing power required for a block matching motion estimation algorithm increases almost linearly with the 

55 number of pixels in the blocks. In the recursive motion estimation algorithm of the article mentioned in the introduction, 
where vectors are determined not only by the pixels in the block, but also by the neighboring blocks, the sensitivity for 
subsampling appeared to be not very high. This was verified with a subjective test and by calculating the average squared 
pixel differences between an interpolated and an original field in a number of test sequences. The influence of subsam- 
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pling on the Mean Square Error (MSE) is small up to a sub'sampling factor of four, as can be seen from table 1. 
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effect of subsampiing factor (S) on matching error (MSE). 











Two further conclusions were drawn from experiments: 

1 . The subsample pattern should cover the largest possible area of the block, 

2. A homogeneous distribution of pixels seems best (a subsample pattern having a denser sampling in the centre 
was tested, but appeared to yield worse results than a homogeneous distribution. 

As covered area seems to be of more importance than the number of pixels in the block, the density of pixels was 
further reduced, while at the same time the block area was extended (so that the amount of pixels in the blocks remains 
the same). One tested subsample pattern covers a block having a size of 16 pixels by 4 lines, each line containing, 
however, only 4 samples in a quincunx subsample pattern. The vectors calculated for this block of 16 x 4 assigned to 
a block of 8 x 4. So the blocks on which the match is calculated, are wider than the blocks to which the vectors are 
assigned, which implies that the information can be subsampled by a factor of four so that the size of the field delay 
furnishing the samples of the preceding field can be reduced by the same factor of four. 

Additional to the hardware advantage this pixel subsampiing and block widening technique appeared to yield in- 
creased quality; the wide blocks with lower density (16 pixels) gave a lower MSE than the normal (8 x 4) blocks without 
subsampiing (32 pixels). 

If this pixel subsampiing and the block subsampiing described hereinbefore are combined, no access is required 
to pixels more than once, which multiple access would be unattractive as it would require a more complex hardware. 

Further it was verified that the result obtained by the combination of pixel subsampiing and block subsampiing is 
better than the result obtained with subsampled wide blocks (16 x 4, 16 pixels) and no block subsampiing. 




Claims 

1. Apparatus for determining motion vectors for each of a plurality of image blocks together constituting a television 
image, characterized by comprising: 

firstmeans (40) for computing first motion vectors only for selected ones (X) of the image blocks, (X, O), whereby 
said selected blocks (X) are interleaved (Fig. 1 , 2) with remaining blocks (O); and 

second means (42-50, 20-38) for interpolating second motion vectors for the remaining blocks (O) based on 
said first motion vectors of the selected blocks (X). 

2. Apparatus as claimed in Claim 1, wherein the selected ones (X) of the image blocks (X, O) are alternate blocks 
along each line of each field. 

3. Apparatus as claimed in Claim 2, wherein the alternate lines in each field have a one block line offset with respect 
to each other. 

4. Apparatus as claimed in Claim 1, characterized in that the second means (42-50, 20-38) comprises means for 
separately computing the X and Y components of the second motion vectors for the remaining blocks (O). 

5. Apparatus as claimed in Claim 1 , characterized in that the second means (42-50, 20-38) computes, for each of the 
remaining blocks (O), the median motion vector component of each of the corresponding first motion vector com- 
ponents (I, G; K, E) of two adjoining blocks (X) and the first motion vector component (H(-1)) of the selected block 
(O) occupying the same position in a previous field. 

6. Method for determining motion vectors for each of a plurality of image blocks (X, O) together constituting a television 
image, characterized in that 

first motion vectors are computed only for selected ones (X) of the image blocks, (X, O), whereby said selected 



EP 0 474 272 B1 

blocks (X) are interleaved (Fig. 1, 2) with remaining Blocks (O); and 

second motion vectors for the remaining ones (O) of the image blocks (X, O) are interpolated from the computed 
first motion vectors of the selected blocks. (X). 

5 7. A method as claimed in Claim 6, characterized in that the interpolation of the second motion vectors comprises the 
step of finding a median of computed first motion vector components (I, G; K, E) in adjoining blocks (X) of the same 
field and a computed first motion vector component (H(-1)) from a previous field. 

8. A method as claimed in Claim 6, characterized in that the number of selected blocks (X) is substantially less than 
10 the total number of image blocks (X, O) of said image. 

9. A method as claimed in Claim 6, characterized in that said first motion vectors are computed on the basis of sub- 
sampled blocks which cover said image blocks, said subsampled blocks having a reduced sampling density in 
comparison with said image blocks, and said subsampled blocks having a size which exceeds a size of said image 

is blocks. 

10. Apparatus for determining motion vectors for each of a plurality of image blocks (X, O) together constituting an 
image, characterized by comprising: 

20 means (40) for receiving first motion vectors computed only for selected ones (X) of the image blocks; (X, O), 

whereby said selected blocks (X) are interleaved (Fig. 1, 2) with remaining blocks (O); and 
means (42-50, 20-38) coupled to said receiving means (40) for interpolating second motion vectors for the 
remaining blocks, (O) basis of said computed first motion vectors of the selected blocks (X). 

25 ^ 
Patentanspruche 

1. Anordnung zur Bestimmung von Bewegungsvektoren fur jeden einer Anzahl Bildblocke, die zusammen ein Fern- 
sehbild formen, dadurch gekennzeichnet , daB sie die nachfolgenden Elemente aufweist: 

30 

erste Mittel (40) zum Berechnen erster Bewegungsvektoren nur fur selektierte Blocke (X) der Bildblocke (X, 
O), wobei die genannten selektierten Blocke (X) mit den restlichen Blocken (O) verschachtelt sind (Fig. 1 , 2), und 
zweite Mittel (42-50, 20-38) zum Interpolieren zweiter Bewegungsvektoren fur die restlichen Blocke (O) auf 
Basis der genannten ersten Bewegungsvektoren der selektierten Blocke (X). 

35 

2. Anordnung nach Anspruch 1 , wobei die selektierten Blocke (X) der Bildblocke (X, O) wechselnde Blocke langs;jeder 
Zeile jedes Teilbildes sind. 

3. Anordnung nach Anspruch 2, wobei die wechselnden Zeilen in jedem Teilbild eine Einblockzeilenversetzung gegen- 
40 Qber einander haben. 

4. Anordnung nach Anspruch 1 , dadurch gekennzeichnet, daB die zweiten Mittel (42-50, 20-38) Mittel aufweisen zur 
einzelnen Berechnung der X- und Y-Anteile der zweiten Bewegungsvektoren der restlichen Blocke (O). 

45 s. Anordnung nach Anspruch 1 1 dadurch gekennzeichnet , da(3 die zweiten Mittel (42-50, 20-38) fur jeden der restli- 
chen Blocke (O) den mittleren Bewegungsvektoranteil jedes der entsprechenden ersten Bewegungsvektoranteile 
(I, G; K, E) zweier benachbarter Blocke (X) und den ersten Bewegungsvektoranteil (H(-1)) des selektierten Blocks 
(O) berechnen, der dieselbe Lage in dem vorhergehenden Teilbild hat. 

so 6. Verfahren zur Bestimmung von Bewegungsvektoren fur jeden einer Anzahl Bildblocke (X, O), die zusammen ein 
Femsehbild formen, dadurch gekennzeichnet , daO 

erste Bewegungsvektoren nur fur selektierte Blocke (X) der Bildblocke (X, O) berechnet werden, wobei die 
genannten selektierten Blocke (X) mit den restlichen Bjdcken (O) verschachtelt werden (Fig. 1 , 2) und 
55 zweite Bewegungsvektoren fur die restlichen Blocke (O) der Bildblocke (X, O) von den berechneten Bewe- 

gungsvektoren der selektierten Blocke (X) interpoliert werden. 

7. Verfahren nach Anspruch 6, dadurch gekennzeichnet . daG die Interpolation der zweiten Bewegungsvektoren den 



5 



EP 0 474 272 B1 

Verfahrensschritt aufweist, durch den ein Mittelwert voh berechneten ersten Bewegungsvektoranteilen (I, G; K, E) 
in benachbarten Blocken (X) desselben Teilbildes und einem berechneten ersten Bewegungsvektoranteil (H(-1)) 
aus einem vorhergehenden Teilbild. 

8. Vertahren nach Anspruch 6, dadurch gekennzeichnet , daB die Anzahl selektierter Blocke (X) wesentlich kleiner 
ist als die Gesamtanzahi Bildblocke (X, O) des genannten Bildes. 

9. Verfahren nach Anspruch 6, dadurch gekennzeichnet. da3 die genannten ersten Bewegungsvektoren auf Basis 
von unterabgetasteten Blocken berechnet werden, welche die genannten Bildblocke bedecken, wobei die genann- 
ten unterabgetasteten Blocke im Vergleich zu den genannten Bildblocken eine verringerte Abtastdichte aufweisen, 
und wobei die genannten unterabgetasteten Blocke eine GroBe haben, die eine GroBe der genannten Bildblocke 
Qberschrertet. 




10. Anordnung zur Bestimmung von Bewegungsvektoren fur jeden einer Anzahl Bildblocke (X, O), die zusammen ein 
Bild formen, dadurch gekennzeichnet . daB sie die nachfolgenden Elemente aufweist: 

Mittel (40) zum Empfangen erster Bewegungsvektoren, berechnet nur fur seleklierte Blocke (X) der Bildblocke 
(X, O), wobei die genannten selektierten Blocke (X) mit restlichen Blocken (O) verschachtelt werden (Fig 1 
2); und 

Mittel (42-50, 20-38), die mit den genannten Empfangsmitteln (40) gekoppelt sind zum Interpolieren zweiter 
Bewegungsvektoren fur die restlichen Blocke (O) auf Basis der genannten berechneten ersten Bewegungs- 
vektoren der selektierten Blocke (X). 
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25 Revendications 

1. Appareil pour la determination de vecteurs de mouvement pour chacun d'une pluralite de blocs d'image constituant 
ensemble une image de television, caracterise en ce qu'il comprend : 

30 un premier moyen (40) pour calculer des premiers vecteurs de mouvement uniquement pour des blocs selec- 

tionnes (X) des blocs d'image (X, O), lesdits blocs selectionnes (X) etant entrelaces (Fig. 1 , 2) avec des blocs 
restants (O), et 

des seconds moyens (42-50, 20-38) pour interpoler des seconds vecteurs de mouvement pour les blocs restants 
(O) sur la base desdits premiers vecteurs de mouvement des blocs selectionnes (X). 

2. Appareil selon la revendication 1, dans lequel les blocs d'image selectionnes (X) parmi les blocs d'image (X, O) 
sont des blocs qui alternent le long de chaque ligne de chaque trame. 

3. Appareil selon la revendication 2, dans lequel les lignes alternees de chaque trame presentent un decalage reci- 
proque d'une ligne de bloc. 

4. Appareil selon la revendication 1, caracterise en ce que les seconds moyens (42-50, 20-38) comprennent des 
moyens pour calculer separement les composantes X et Y des seconds vecteurs de mouvement pour les blocs 
restants (O). 

5. Appareil selon la revendication 1 , caracterisS en ce que les seconds moyens (42-50, 20-38) calculent, pour chacun 
des blocs restants (O), la composante moyenne de vecteur de mouvement de chacune des premieres composantes 
des vecteurs de mouvement correspondante (I, G; K, E) de deux blocs voisins (X) et la premiere composante du 
vecteur de mouvement (H(-1)) du bloc selectionne (O) occupant la meme position dans une trame precedente. 

6. Precede de determination de vecteurs de mouvement pour chacun d'une pluralite de blocs d'image (X, O) constituant 
ensemble une image de television, caracterise en ce que : 

des premiers vecteurs de mouvement sont calcules uniquement pour des blocs selectionnes (X) des blocs 
d'image (X, O), que lesdits blocs selectionnes (X)-6tant entrelaces (Fig. 1, 2) avec les blocs restants (O), et 
des seconds vecteurs de mouvement pour les blocs restants (O) des blocs d'image (X, O) sont interpoles a 
partir des premiers vecteurs de mouvement calcules des blocs selectionnes (X). 
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7. Proced6 selon la revendication 6, caracterise en ce que f interpolation des seconds vecteurs de mouvement com- 
prend l'6tape de recherche d'une moyenne des premieres composantes de vecteurs de mouvement calcuiees (I, 
G; K, E) dans des blocsVoisins (X) de la m§me trame et d'une premiere composante de vecteur de mouvement 
calculee (H(-1 )) d'une trame precedente. 

5 

8. Proc6d6 selon la revendication 6, caracterise en ce que le nombre de blocs selectionnes (X) est sensiblement 
inferieur au nombre total de blocs d'image (X, O) de ladite image. 

9. Proced6 selon la revendication 6, caracterise en ce que lesdits premiers vecteurs de mouvement sont calcules sur 
10 la base de blocs sous-echantillonnes qui couvrent lesdits blocs d'image, lesdits blocs sous-echantillonnes ayant 

une densite d'echantillonnage reduite en comparaison desdits blocs d'image, et lesdits blocs sous-echantillonnes 
ayant une dimension qui depasse une dimension desdits blocs d'image. 

10. Appareil de determination de vecteurs de mouvement pour chaque bloc d'une pluralite de blocs d'image (X, O) 
is constituant ensemble une image, caracterise en ce qu'tl comprend : 

des moyens (40) pour recevoir des premiers vecteurs de mouvement calcules uniquement pour des blocs 
selectionnes (X) des blocs d'image (X, O), lesdits blocs selectionnes (X) etant entrelaces (Fig. 1 , 2) avec des 
blocs restants (O), 

20 et des moyens (42-50, 20-38) couples auxdits moyens de reception (40) pour interpoler les seconds vecteurs 

de mouvement pour les blocs restants (O) bases sur les premiers vecteurs de mouvement calcules des blocs 
selectionnes (X). 

25 
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